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Abstract — We introduce a thermodynamic (large 
deviation) formalism for computing error exponents 
in error-correcting codes. Within this framework, we 
apply the heuristic cavity method from statistical me- 
chanics to derive the average and typical error expo- 
nents of low-density parity-check (LDPC) codes on 
the binary erasure channel (BEC) under maximum- 
likelihood decoding. 

I. Introduction 

Assessing the performance of error-correcting codes is 
a founding topics of information theory. Amongst the 
simplest codes are the binary block codes, where a source 
generates with equal probability one of 2 L codewords, 
each a sequence of N bits. As a codeword is transmitted 
through a discrete memoryless channel, a noise £ alters 
independently each bit with some probability. The bi- 
nary erasure channel (BEC), for instance, erases a bit 
with a prescribed probability p € [0,1]. Given the re- 
ceived message, the decoding task consists in inferring 
the most likely original codeword. The probability of er- 
ror Pf (error|Cjv) then provides a simple characterization 
of the performance of a code Cn- 

The properties of error-correcting codes are conve- 
niently studied through ensembles of codes Cn, consisting 
for instance of the set of all block codes with length TV 
and rate R = L/N. Shannon showed that, in the limit 
N — > oo, a typical code in such an ensemble has a vanish- 
ing probability of error if (and only if) R < R c (p), where 
R c {p) corresponds to the channel capacity. This capac- 
ity is simply R c (p) = 1 — p for the BEC. We are here 
interested in refining the description of the error proba- 
bility beyond the channel capacity. Error exponents give 
the exponential rate of decay of P^ (error |Cjv) with N, for 
Cn E Cm, and offer the most appealing generalization. 
Of particular interest is the so-called reliability function, 
which gives the lowest achievable exponents as a function 
of the rate R |2]. However, despite significant efforts to 
estimate error exponents, resulting in the establishment 
of a number of bounds, exact expressions are scarce and 
restricted to a few extreme cases. 

In this note, we put forward a thermodynamic (or large 
deviation) formalism for evaluating error exponents 
in error-correcting codes. This formalism coherently en- 
compasses two types of exponents: if C = {Cn}n>i dc- 
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notes a sequence of ensembles of codes, we can indeed de- 
fine, depending on the procedure for choosing the codes 
in the ensembles Cn, an average and a typical error 
exponents as 

E av = - lim i-logE Cjv [P c (error|CAr)], (1) 

iV— >oo iv 

E typ = - lim 1e Cn [logP^errorlCjv)] , (2) 

iV— >oo IX 

where Ec N denotes the expectation value when Cn is 
drawn uniformly from the ensemble Cn (log is base 2 
throughout). Although the typical error exponent is the 
most interesting from the practical point of view, the av- 
erage error exponent is usually simpler to estimate theo- 
retically. 

We analyze in the thermodynamic formalism one of 
the most promising family of block codes, the low-density 
parity-check (LDPC) codes [5]. The codewords of these 
codes correspond to the kernel of a sparse M x N parity- 
check matrix A, with M = N — L. Different choices for 
A lead to different ensemble of codes Cn, the simplest 
example being regular ensembles 1 defined with A having 
£ l's per column and k per line, and zeros otherwise (in 
which case R = 1 — £/k). LDPC codes have been shown 
to formally map to physical models of disordered systems 
on random graphs p], and we shall exploit this analogy 
to apply the (non-rigorous) cavity method ^21 recently 
proposed in this context 2 (see also for a related ap- 
proach) . 

II. Thermodynamic formalism 
Given a received word, consisting of a codeword from a 
code Cn altered by a noise £ on the BEC, let A/tv(£, Cat) be 
the number of codewords from which it could come from 
(this quantity is independent of the initial codeword with 
LDPC codes). By definition, decoding is achievable if 
and only if Nn(£,Cn) = 1- For random codes, the geom- 
etry of the space of codewords indicates that, at least in 

In this paper we restrict to regular codes, even though our 
method can be generalized to any irregular ensemble 111) . 

2 While the exponential scaling of the error probability is guar- 
anteed when the ensemble of codes comprises all block codes, the 
average error probability of LDPC codes is known to be polyno- 
mial in N HJ- Following Gallager, we shall ignore the few atypical 
codes responsible for this behavior, and consider the average error 
exponent associated with an expurgated ensemble where they have 
been excluded 0. 



the vicinity of the channel capacity, an error most prob- 
ably involves an exponential number of potential code- 
words (see e.g. PP). In such situations, we characterize 
jVjv(C)^Jv) by an entropy, defined as 



S N (Z,C N )=logSS N (£,C N ). 



(3) 



In the limit N — > oo, for sequences of codes C = {Cn}n 
taken from the sequence of ensembles C = {Cn}n, the 
entropy density s = Sn/N concentrates to a well defined 
value s, and the channel coding theorem takes the follow- 
ing form: there exists p c , such that s — for p < p c , and 
s > for p > p c 0j . More generally, we postulate that, 
for a typical sequence of codes C° = {C%}n, the entropy 
Spj satisfies a large deviation principle [2], i.e., 

si x 2~ Jvio(s) 



Ft[S N (£,C u N )/N 



(4) 



with ajv 5< 6jv meaning that logajv/ log&jv — * 1- The 
typical value s corresponds here to the minimum of the 
rate function Lq, with Lo(s) = 0. In cases where Lq is 
strictly convex, the typical error exponent is obtained as 



E typ = - km I log VdS N ^C° N )/N = s] 

N— >oc iV z — 



s>l/N 



(5) 



£o(* = 0). 



A simpler quantity to compute than L (s) is -Ea(s), the 
rate function for the large deviations of Sn(£,Cn) with 
respect to both the noise £ and the codes Cjv, 

2 -A r L 1 (.s) 



■,c J ,[5 JV (^C JV )/JV = «] 



(6) 



In the so-called thermodynamic formalism Li(s) is 
associated with a potential cf>(x) defined through the re- 
lation 



ds 2 



JV[. 



(7) 



Under the assumption that it is convex, the rate function 
Li(s) is derived from the knowledge of 4>(x) by Legendre 
transformation: 



Li(s) = max [xs — <fi(x)] 



(8) 



The average exponent, obtained from 75 av = L\(s = 0), 
may differ from the typical exponent .Etyp- Typical codes 
C^q can however also be described within a thermody- 
namic formalism, provided an extra "temperature" y is 
introduced, together with a generalized potential ip(x,y) 
satisfying 



(E C [2 



xS N (£,C N ) 



(9) 



The average case is here recovered for y = 1, with 
ip(x,y = 1) = </>(x). Typical error exponents are asso- 
ciated with y = (see for details and exceptions), 
with 



E 



typ 



£o(* = 0) 



where x* selects for s = ^d x ip(x* ,y) 



d y i>{x*,y = Q), 
= 0. 



(10) 



y=o 



III. Cavity method 

Disordered systems constructed out of random ensem- 
bles, of which LDPC codes are particular examples, have 
been the subject of intensive studies in statistical me- 
chanics. One of the most elaborate analytical tool de- 
veloped in this context is the cavity method |10j . which 
allows to extract the typical properties of models defined 
on random graphs. While yielding virtually equivalent 
predictions than the similar replica method, this method 
has both more sound probabilistic foundations, and an 
attractive relation to message-passing algorithms, such 
as belief propagation (BP). The cavity method has also 
been recently extended to deal with large deviations J2| > 
making it perfectly suited to the evaluation of error ex- 
ponents. 

As far as typical codes and typical noise are concerned, 
the cavity method is equivalent to a BP density evolution 
analysis. Belief propagation, also known as the "peeling 
decoder" in the context of the BEC [8], consists in prop- 
agating messages between bits (the N letters of a word) 
and checks (the M linear equations encoded in the parity- 
check matrix A that each codeword must satisfy). The 
messages can take three different values: * (erasure) or 
or 1. Initially, each bit sends its value or 1, or * if erased, 
to each of the parity checks it is involved in. Chcck-to-bit 
and bit-to-check messages are then sent alternatively. If 
a check a receives non-erasure messages from all its bits 
but i, it sends to i the sum (modulo 2) of these messages; 
otherwise, the check a sends * to i. If an erased bit i 
receives at least one non-erasure message from any of its 
checks but a, it sends it to a (if more than one, they are 
necessarily identical); otherwise, the bit i sends its value, 
or 1, or * if erased, to a. The algorithm stops after 
convergence of the iterations. 

The (typical) cavity method, or BP density evolution, 
analyzes the outcome of this procedure in the limit where 
the codeword length TV is infinite. It introduces rj, the 
probability that a bit sends an erasure message to a check, 
and C the probability that a check sends an erasure mes- 
sage to a bit, both taken after BP has reached conver- 
gence. The cavity equations satisfied by these two proba- 
bilities, 



C = i - (i - v) 



fc-i 



v = K 



(ii) 



characterize the fixed point of the BP density evolution 
(see Fig.nj. 

Once BP has converged, bits receiving at least one non- 
erasure message are fixed to their correct value, as are 
the non-erased bits. When eliminated, along with the 
checks receiving no more than one erasure message, they 
leave the so-called core. The dimensions M c x 7V C of the 
associated residual matrix are, with high probability: 



N c =pCN + o(N), 

M c = -[!-(!- n) k - k V (l - itf-^N + o(N). 



(12) 



(a) 




<b) ' >Vf 



Figure 1: Illustration of the cavity equations with 
k = 4 and ^ = 3. (a): a check node (square) sends an 
erasure message to a bit node (dashed circle) if at least 
one of its other variables sends an erasure message, (b): 
a bit node (circle) sends an erasure message to a check 
node (dashed square) if it has been erased and if all its 
other checks send an erasure message. 



For p < Pd(£, k), the only solution to is C = 0, ?/ = 0, 
meaning that BP is able to decode the whole word with 
high probability. For p > Pd however, BP gets stuck at 
some £ > 0, T) > 0. In this case, it can be proved that 
the residual matrix has full-rank with high probability 
|2J- Therefore, the problem has exactly 2 Nc ~ Ma solutions 
if N c > M c , and one solution (the original codeword) 
otherwise. In this approach, the critical noise p c {£, k) is 
obtained from the condition N c = M c , and s is given by 
max(0, s cav ), with s cav = lirn/v^oo (Nc - M c )/N. 

The large deviation cavity method is built on the same 
ideas but incorporates a biased measure over the noise 
and code ensemble, as prescribed by Eq. When we 
consider the value of a bit-to-check message as a function 
of its (£ — l)(fc — 1) "grandparents" , we also evaluate the 
"entropy shift" AS associated with the addition of the 
bit and its £ — 1 checks, i.e. the difference between the 
numbers of columns and lines contributed by the bit and 
its checks to the residual matrix. Then the message is 
sent with a probability proportional to 



(E 5 2 



')• (13) 
For regular LDPC codes, we thus obtain for the potential 



tp(x,y) = 

\ogZ t - e -^-^ log [(1 - „)* + (!-(!- V ) k )2^y] 



(14) 



with 



Z e = {(2-' xy + 1-CY -{C2- X ' V Y + C l {pT + l-p) y 2- ixy 

(15) 

and 



r] = cY~ 1 {p2 x ) y 2~^- 1 >yz-\, 
C = i-(i- v ) k - 1 . 



(16) 



Note that the entropy conjugated with x is not the "real" 
entropy s, but s cav = (N c — M c )/N. When x = 0, the 
fixed point of the usual density evolution equations i|llf) 
is recovered, with {l/y)d x '^{x = 0, y) giving back s cav , 
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Figure 2: Average entropic rate function Li(s) as a func- 
tion of the entropy density s C av, for the regular LDPC 
code £ = 3, k = 6 on the BEC with increasing values 
of p. The real entropy is actually s = max(0, s cav ). (a): 
P < Pirsb, no solution with s = 0; (b): pi rs b < p < Pd, a 
solution with s = 0, but s is not defined; (c): pd < p < p c , 
s = 0; (d): p > p c , s > indicates that decoding typically 
fails. 



the typical value. 



IV. LDPC CODES 

We first discuss average error exponents. The calcula- 
tion of the average rate function Li(s) reveals four dis- 
tinct regimes when the noise level p is varied, as illus- 
trated and explained in Fig. [21 In particular, we find 
that the rate function £i(s) is no longer defined for s = 
when p is too small (p < pi rs b), which points to the inad- 
equacy of our method in this low-noise regime. 

Indeed, by retaining s = as criterion for correct de- 
coding, we assumed that an error implicates an exponen- 
tial number of codewords. An error may however also be 
caused by the presence of one (or a few) isolated code- 
word^). Estimating this probability requires an alterna- 
tive, "energetic", scheme, as opposed to the "entropic" 
scheme discussed so far 3 . Equations for the energetic av- 
erage and typical error exponents can also be obtained 
from the large deviation cavity method [llj . but their 
solutions are confined to a restricted interval p > p TS , in- 
dicating again that the lowest noise levels are not appro- 
priately described. The entropic and energetic exponents 
are found to cross at p e , which corresponds to the so- 
called critical rate We conjecture that the entropic 



3 The energetic version of the cavity method is also referred to 
as "replica symmetric" in the physics literature, while the entropic 
version is known as "one-step replica symmetry breaking" . 
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Figure 3: Average error exponent as a function of the 
noise level p of the BEC for the regular LDPC code en- 
semble with k = 6 and £ = 3. Gallager's union bound 
and the random linear code limit l|19J) are also plotted for 
comparison. 



(k,£) 


(4,3) 


(6,3) 


Plrsb 


0.3252629709 


0.2668568754 


Pis 


0.5465748811 


0.3378374641 


Pe 


0.6068720166 


0.3491884902 


Pd 


0.6474256494 


0.4294398144 


Pc 


0.7460097025 


0.4881508842 



Table 1: Thresholds pi Is b, Pvs, Pe, Pd and p c (see text 
and Fig. EJ) for two regular ensembles of LDPC codes. 



exponent, as given by the above equations, is exact in the 
range [p e ,p c ], while the energetic exponent (not presented 
here), which applies for [p rs ,p e ], is only approximate. 

Fig. El shows our predictions for the average exponent 
of the i = 3, k = 6 regular LDPC codes, with the two 
regimes represented; the same general picture holds for 
other regular or irregular ensembles (see also Tabic 0. 

V. The Random Linear Code limit 
This limit is obtained from regular codes with fc, £ — > oo 
and R = 1 — l/k fixed, where the potential simplifies to: 



ip(x, y)=y \og{p2 x + l-p) + (R - l)xy. 



(17) 



The trivial dependence of ip(x, y) with y implies that the 
two error exponents E av and E typ , as obtained from the 
cntropic scheme, are identical. They are equal to the vol- 
ume bound [2] D(l — R\ \p), where D(x\ \y) = x \og(x/y) + 
(1 — x) log((l — x)/(l — y)) denotes the Kullback-Leibler 
divergence. 

The intersection of the entropic and energetic average 
error exponents yields the threshold 



1-R 
1 + R' 



(18) 




Figure 4: Average and typical error exponents of random 
linear codes on the BEC as a function of p, with R = 1/2 
fixed. Inset: the same exponents as a function of R, with 
p = 0.4 fixed. 



and we obtain for the average error exponent in the infi- 
nite connectivity limit: 



^av(RLC) 



1 - R - log(l 
D(l-R\\p) 



P) 



if p < p e , 

if p e < p < p c . 



(19) 

It coincides with the average error exponent of the ran- 
dom linear code (RLC) ensemble, where the MxN parity- 
check matrix is chosen at random with uniform probabil- 
ity among all possible parity-check matrices. Assuming 
that the inversion of the limits N — » oo and k,£ — > oo 
is justified, we interpret this result as a validation of our 
approach (note that here, p rs = 0). 

The analysis of the typical error exponent in the ener- 
getic regime leads us to introduce an additional threshold, 



Py 



JGV 



(R) 



1 



Sov ( R y (20 » 

where Sgv{R), the minimal reduced distance of a typ- 
ical linear code [Q, is given by the smallest solution of 
—5 log 5 — (1 — S) log(l — S) = 1 — R. Below p v , physical 
arguments indicates that the typical error exponent 
must differ from the average one, with: 



Btyp(RLC) — 



-Sgv(R) logp 

^av(RLC) 



if P<Py, 

\fp>Py. 



(21) 



We are not aware of any previous report of this expression 
in the literature, but the fact that it matches the union 
bound suggests that it is exact. Fig. 0] presents the error 
exponents as a function of p for a fixed value of the rate 
R = 1/2. 

The two thresholds p e and p y are presumably generic 
features of block codes, and are also found with random 
codes on the binary symmetric channel pQ. 



VI. Discussion 

Despite being one of the earliest and most basic topics 
in information theory, error exponents still retain today 
a number of unsolved issues. We advocated here a novel, 
thermodynamical, formulation of this problem. Using the 
cavity method from statistical mechanics, we worked out 
in this framework expressions for the average and typical 
error exponents of LDPC codes on the BEC. 

While non rigorous, the cavity method aims at pro- 
viding exact formulae. Accordingly, our expressions are 
consistent with the various rigorous studies reported in 
the literature. The quest for rigorous proofs of formula) 
obtained from the cavity method is currently an active 
fields of mathematics |15) . Remarkably, predictions from 
the cavity method on the maximum-likelihood threshold 
Pc 01 could be turned into rigorous theorems (Hj- This 
may inspire alternative derivations of our results. 

Perhaps not too surprisingly, the entropic range p e < 
p < p c where we conjecture our results to be exact also 
coincides with the limited interval for which the related 
problem of determining the reliability function of block 
codes has been solved so far. Extending our method to 
p < p e , where we could obtain only approximate results 
(except in the infinite connectivity limit), remains a chal- 
lenging open problem. 

Using the same approach, we also analyzed the case of 
the binary symmetric channel, obtaining comparable re- 
sults ■ A more interesting extension would be to iter- 
ative decoding, such as BP. Although arguably quite aca- 
demic, studying maximum-likelihood decoding, as we did, 
is nevertheless certainly an essential preliminary step. 
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